Stateless stream handling and resharding

ABSTRACT

Systems and methods are disclosed for stateless stream handling and resharding. In one implementation, a first shard including one or more messages is generated. The first shard is associated with a first shard version attribute. The first shard and the first shard version attribute are provided as a first atomic update within a data stream. The first shard is resharded into at least a second shard. The second shard is associated with a second shard version attribute. The second shard and the second shard version attribute are provided as a second atomic update within the data stream.

TECHNICAL FIELD

Aspects and implementations of the present disclosure relate to dataprocessing and, more specifically, but without limitation, to statelessstream handling and resharding.

BACKGROUND

Streaming systems can include devices that provide or push data on aregular basis. Other devices may request or pull this data, e.g., inorder to process it.

SUMMARY

The following presents a shortened summary of various aspects of thisdisclosure in order to provide a basic understanding of such aspects.This summary is not an extensive overview of all contemplated aspects,and is intended to neither identify key or critical elements nordelineate the scope of such aspects. Its purpose is to present someconcepts of this disclosure in a compact form as a prelude to the moredetailed description that is presented later.

In one aspect of the present disclosure, systems and methods aredisclosed for stateless stream handling and resharding. In oneimplementation, a first shard comprising one or more messages isgenerated. The first shard is associated with a first state attribute.The first shard and the first state attribute are provided as an updatewithin a data stream.

In another aspect of the present disclosure, a first shard including afirst state attribute is received within a first stream. A message thatis inconsistent with the first state attribute is identified within thefirst shard. The message is associated as an attribute of the firstshard. A second shard including a second state attribute is received.Based on the second state attribute, a position of the message withinthe second shard is determined. The message is inserted into the secondshard based on the determining.

In another aspect of the present disclosure, a first shard including oneor more messages is received. The first shard is associated with a firststate attribute. The first shard and the first state attribute areprovided as an update within a data stream.

In another aspect of the present disclosure, a first shard including oneor more messages is generated. The first shard is associated with afirst shard version attribute. The first shard and the first shardversion attribute are provided as a first update within a data stream.The first shard is resharded into at least a second shard. The secondshard is associated with a second shard version attribute. The secondshard and the second shard version attribute are provided as a secondupdate within the data stream.

In another aspect of the present disclosure, a first shard including oneor more messages and a first shard version attribute is received from adevice. A current shard version is requested from the device. Based on adetermination that the current shard version is consistent with thefirst shard version attribute, an operation is performed with respect tothe first shard.

In another aspect of the present disclosure, a first shard including oneor more messages and a first shard version attribute is received. Acurrent shard version is requested. Based on a determination that thecurrent shard version is consistent with the first shard versionattribute, an operation is performed with respect to the first shard.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects and implementations of the present disclosure will be understoodmore fully from the detailed description given below and from theaccompanying drawings of various aspects and implementations of thedisclosure, which, however, should not be taken to limit the disclosureto the specific aspects or implementations, but are for explanation andunderstanding only.

FIG. 1A illustrates an example system, in accordance with an exampleembodiment.

FIG. 1B illustrates an example system, in accordance with an exampleembodiment.

FIG. 2A is a flow chart illustrating a method, in accordance with anexample embodiment, for stateless stream handling and resharding.

FIG. 2B is a flow chart illustrating a method, in accordance with anexample embodiment, for stateless stream handling and resharding.

FIG. 3 is a flow chart illustrating a method, in accordance with anexample embodiment, for stateless stream handling and resharding.

FIG. 4A illustrates an example system, in accordance with an exampleembodiment.

FIG. 4B illustrates an example system, in accordance with an exampleembodiment.

FIG. 5A is a flow chart illustrating a method, in accordance with anexample embodiment, for stateless stream handling and resharding.

FIG. 5B is a flow chart illustrating a method, in accordance with anexample embodiment, for stateless stream handling and resharding.

FIG. 5C is a flow chart illustrating a method, in accordance with anexample embodiment, for stateless stream handling and resharding.

FIG. 6 is a block diagram illustrating components of a machine able toread instructions from a machine-readable medium and perform any of themethodologies discussed herein, according to an example embodiment.

DETAILED DESCRIPTION

Aspects and implementations of the present disclosure are directed tostateless stream handling and resharding.

As described herein, various device(s), system(s), etc. can generatedata, content, commands, etc., such as such as messages or events. Incertain implementations, such messages, commands, events, etc. can bestructured, formatted, provided and/or transmitted in various ways, suchas a stream, feed, queue, etc. Examples of such device(s) (from whichmessages, events, etc., originate) include but are not limited to:computing devices, Internet of Things (‘IoT’) devices, sensors, systems,other devices, services, and/or functions, and/or any other element orsource capable of generating, providing, and/or otherwise makingaccessible the messages, commands, events, data, etc., described herein.In various examples illustrated here, the referenced device(s) (fromwhich a stream of messages, events, etc. can originate) may be referredto as “producer(s).”

As also described herein, various device(s), system(s), etc. can beconfigured to access, analyze, process, and/or perform various otheroperations on data, content, commands, etc., such as such as messages orevents (e.g., stream(s), feed(s), etc. of messages, events, etc.,originating from the producer(s) referenced above). Examples of suchdevice(s) or system(s) (that process the referenced streams) include butare not limited to: computing devices, systems, services, and/or anyother element capable of processing and/or otherwise performingoperations with respect to the streams, messages, commands, events,data, etc., described herein. In various examples illustrated here, thereferenced device(s) (that can process streams of messages, events,etc.) may be referred to as “consumer(s).”

Various modern systems may employ multiple producers and multipleconsumers in various topologies or arrangements. Such streaming systemsmay, for example, be configured to ensure that all events, messages,etc., within a stream are handled (e.g., provided by a producer and/orprocessed by a consumer) at least once. In scenarios in which multipleproducers and/or consumers are present, such streaming systems may beconfigured to provide certain messages, events, etc. multiple times,such as in the event of a malfunction, crash, failure, etc., at aproducer. In such a scenario, various messages, events, etc., may beprovided multiple times, and it may be necessary to identify and/orresolve such redundancy (e.g., by the consumer when processing thereferenced messages, events, etc.).

It can therefore be appreciated that various inefficiencies are presentin streaming systems or services configured to stream and/or processeach event, message, etc., within a stream ‘at least once.’

Accordingly, described herein are technologies that enable streamingsystems/services to provide and/or process such events or messages once(e.g., ‘exactly once) and avoid redundancies or inefficiencies even inscenarios in which a producer or consumer fails. In doing so, thedescribed technologies maintain the resiliency of a streaming system andenable stream producers and consumers to recover from failures whileensuring ‘exactly once’ semantics. Additionally, the describedtechnologies can enable conditional updates and stateless operations, asdescribed herein.

It can therefore be appreciated that the described technologies aredirected to and address specific technical challenges and longstandingdeficiencies in multiple technical areas, including but not limited tocontent streaming, content delivery, and data processing. As describedin detail herein, the disclosed technologies provide specific, technicalsolutions to the referenced technical challenges and unmet needs in thereferenced technical fields and provide numerous advantages andimprovements upon conventional approaches. Additionally, in variousimplementations one or more of the hardware elements, components, etc.,referenced herein operate to enable, improve, and/or enhance thedescribed technologies, such as in a manner described herein.

By way of illustration, FIG. 1A depicts an example system 100, inaccordance with some implementations. As shown in FIG. 1A, system 100can include devices such as device 110A and device 110B (also referredto herein as ‘producer(s)’), as well as other systems, services,entities, etc., as described herein. Various devices can be connected toand/or otherwise communicate or transmit information, data, etc., to oneanother via various networks, connections, protocols, etc. (e.g., viathe internet).

The referenced producers (e.g., device 110A as shown in FIG. 1A) can be,for example, a server computer, computing device, storage service (e.g.,a “cloud” service), etc. which a stream of messages, events, etc. canoriginate. In certain implementations, such devices can include streamproduction engine 112.

Stream production engine 112 can be a program, module, or set ofinstructions that configures/enables a device (e.g., a producer such asdevice 110A as shown in FIG. 1A) to perform various operations such asare described herein. Such instructions, etc., can be stored in memoryof the device (e.g. memory 630 as depicted in FIG. 6 and describedbelow). One or more processor(s) of the device (e.g., processors 610 asdepicted in FIG. 6 and described below) can execute such instruction(s).In doing so, the device can be configured to perform various operations,such as those described herein. For example, stream production engine112 can configure the device to generate shard(s) and/or perform otheroperations as described herein.

As also shown in FIG. 1A, system 100 can also include devices such asdevice 110C and device 110D (also referred to herein as consumer(s)).Such devices can be, for example, a server computer, computing device,services (e.g., a “cloud” service), etc. configured to access, analyze,process, and/or perform various other operations on messages or events(e.g., stream(s), feed(s), etc. originating from the producer(s)referenced above). In certain implementations, such devices can includestream consumption engine 114. Stream consumption engine 114 can be aprogram, module, or set of instructions that configures/enables a device(e.g., device 110C as shown in FIG. 1A) to perform various operationssuch as are described herein. For example, stream consumption engine 114can configure the device to request and/or process messages, events,etc., such as those originating from ‘producer’ devices, as describedherein.

Additionally, in certain implementations system 100 can also server 120.Server 120 can be, for example, a server computer, computing device,services (e.g., a “cloud” service), etc. configured to manage variousaspects of a distributed streaming system (e.g., a system thatincorporates multiple producers and/or consumers. In certainimplementations, server 120 can include repository 122 and/or streammanagement engine 124. Repository 122 can be, for example, variousstorage resource(s) such as an object-oriented database, a relationaldatabase, memory, etc. with respect to which data (e.g., shards,messages, objects, etc., such as those referenced herein) can beretrieved and/or stored. Stream management engine 124 can be a program,module, or set of instructions that configures/enables server 120 toperform various operations such as are described herein. For example,stream management engine 124 can configure server 120 to update (and/orperform various other operations or transformations on) a record, shard,message, object etc. stored in repository 122, as described herein.

Further aspects and features of system 100 are described in more detailbelow.

As used herein, the term “configured” encompasses its plain and ordinarymeaning. In one example, a machine is configured to carry out a methodby having software code for that method stored in a memory that isaccessible to the processor(s) of the machine. The processor(s) accessthe memory to implement the method. In another example, the instructionsfor carrying out the method are hard-wired into the processor(s). In yetanother example, a portion of the instructions are hard-wired, and aportion of the instructions are stored as software code in the memory.

FIG. 2A is a flow chart illustrating a method 200, according to anexample embodiment, for stateless stream handling and resharding. Themethod is performed by processing logic that can comprise hardware(circuitry, dedicated logic, etc.), software (such as is run on acomputing device such as those described herein), or a combination ofboth. In one implementation, the method 200 is performed by one or moreelements depicted and/or described in relation to FIG. 1A (including butnot limited to device 110A and/or stream production engine 112), whilein some other implementations, the one or more blocks of FIG. 2A can beperformed by another machine or machines.

For simplicity of explanation, methods are depicted and described as aseries of acts. However, acts in accordance with this disclosure canoccur in various orders and/or concurrently, and with other acts notpresented and described herein. Furthermore, not all illustrated actsmay be required to implement the methods in accordance with thedisclosed subject matter. In addition, those skilled in the art willunderstand and appreciate that the methods could alternatively berepresented as a series of interrelated states via a state diagram orevents. Additionally, it should be appreciated that the methodsdisclosed in this specification are capable of being stored on anarticle of manufacture to facilitate transporting and transferring suchmethods to computing devices. The term article of manufacture, as usedherein, is intended to encompass a computer program accessible from anycomputer-readable device or storage media.

At operation 205, a first shard is generated. In certainimplementations, such a shard can be an object or partition (e.g., froma larger database or object) and can include one or more messages,events, records, etc., as described herein. For example, as shown inFIG. 1A, shard ‘S1’ (130A) can be generated by producer 110A (e.g., bystream production engine 112). As shown in FIG. 1A, shard 130A caninclude messages 132 (e.g., messages ‘M1,’ ‘M2,’ ‘M3,’ etc.).

At operation 210, the first shard (e.g., as generated at operation 205)is associated with an attribute. In certain implementations, such anattribute can be a state attribute 152A, such as state attribute(“STATE”) ‘X1,’ as shown in FIG. 1A. Such a state attribute can reflect,for example, various aspects of the state of the producer. Examples ofsuch a state can include but are not limited to a quantity or valuecorresponding to the number of messages being produced by the producer(e.g., per second), various aspects of data transformation beingperformed by the producer, and/or other state(s) reflecting the statusor operation(s) of the producer.

In certain scenarios, storing/associating a state attribute with a shardcan enable a producer that fails or malfunctions to be reinitialized andcontinue providing shards, messages, records, etc., within a stream. Forexample, as described herein, in a scenario in which a producer failsand is reinitialized, the producer can request (e.g., from server 120and/or repository 122) the state attribute (e.g., from a shardassociated with the same producer and provided to/received by server120). In response, the producer can receive a state attribute/identifierthat reflects, for example, the state of the producer (e.g., when themost recently received shard was provided). The producer can thenreinitialize and continue providing shards, messages, etc. based on suchreceived state (rather than, for example, providing redundant copies ofshards/records that have been previously received by the streamingsystem). It can be appreciated that such a configuration can enable theproducer to operate in a stateless manner.

Additionally, as shown in FIG. 1A, in certain implementations thedescribed technologies can further assign or associate additionalattributes (e.g., attribute 152B) to the referenced shards, messages,etc. Such attributes can, for example, enable various entities,services, systems, etc. (e.g., server 120) to collect, monitor, and/orgenerate various metrics, statistics, etc., that reelect aspects of theoperation of a producer. By way of illustration, such attributes (whichcan be associated to a shard, e.g., by the producer from which itoriginates) can reflect the number of messages, events, etc. pushed bythe producer, number of records updated, messages since a last pushoperation, various latencies associated with operations of the producer(e.g., push latency), etc. In doing so, the streaming system can monitorthe operation of various producer(s), and can further adjust variousother operations based on the referenced metrics, statistics, etc., asdescribed herein.

Additionally, in certain implementations such a state attribute canreflect an importance and/or location of one or more of the messages(e.g., within the associated shard/stream). By way of illustration, in ascenario in which messages, data, etc., being provided/pushed by theproducer into the stream has a structured format, the describedtechnologies (e.g., stream production engine 112) can enable variousoperations/transformations. For example, an attribute (e.g., attribute152B as shown in FIG. 1A) and/or other such property of a shard (e.g.,shard ‘S1’) can be assigned/updated based on message(s), data, etc.,within the shard. By way of illustration, such attribute(s),propertie(s), etc. can be used for statistics (e.g., reflecting messageproperties such as message types), alerts (e.g., based on content of amessage within the stream), location markers (e.g., reflecting locationof certain messages within a shard), etc.

By way of further illustration, it can be appreciated that certainmessages provided/pushed by a producer may be of particularsignificance, importance, etc. (e.g., messages containing certaincontent). Accordingly, it can be advantageous to configure the describedtechnologies to enable such message(s) to be easily accessed,identified, etc. In certain implementations, when generating a shardthat includes message(s) of particular importance (e.g., messagescontaining certain types of content), an attribute 152B or other suchproperty can be associated with the shard, reflecting that it containsan important message. Upon receiving such a shard (with the referencedattribute/property), streaming system 120 and/or consumer 110C canprioritize the processing, analysis, etc. of such a shard/message(and/or perform other operations). By way of further example, thereferenced attribute 152B or property can reflect the location (e.g.,within the shard) of such important, significant, etc. message(s). Indoing so, the state/attributes of the shard can reflect content withinits messages and can further enable operations to be performed on suchmessages (e.g., in a prioritized manner).

Additionally, in certain implementations, the referenced attribute 152B(which can be used to adjust/control operation of the producer) caninclude/reflect a token, such as may be assigned based on a processingcapacity of a streaming system. Such a token may be assigned (e.g., to ashard, message, etc.) based on a processing capacity of a streamingsystem. In certain implementations, such a token can be assigned by thestreaming system 120 and/or by a consumer (e.g., consumer 110C) of thestream. For example, the referenced tokens can be used to implement flowcontrol operations which can, for example, adjust operation of theproducer (e.g., in scenarios in which shards, messages, etc., are beingprovided too quickly). Further aspects of the referenced flow controloperations are described below, e.g., at operation 235.

In other implementations, the referenced attribute can include orreflect an identifier such as a sequence identifier. Such a sequenceidentifier can reflect the position of the associated shard (and/ormessage(s)) within a sequence. By way of illustration, a time/date stamp(reflecting, for example, the time/date the associated shard,message(s), etc., was/were received, created, and/or provided) can beused as a sequence identifier. In doing so, the relative position of acertain shard can be determined. For example, a sequence identifierassociated with shard ‘S2’ (as shown in FIG. 1A) can reflect that such ashard was received, created, and/or provided after shard ‘S1’ and beforeshard ‘S3’.

It should be noted that in scenarios in which multiple producers arepresent (e.g., as shown in FIG. 1A), the referenced attribute(s) canfurther include a field, identifier, etc., that reflects the producerfrom which the associated shard/message(s) originated. Accordingly, inthe scenario depicted in FIG. 1A, shard(s) originating from producer110A can be associated with an attribute reflecting the identity of theproducer and a timestamp, while shard(s) originating from producer 110Bcan be associated with corresponding identifier(s) also reflecting theidentity of that producer (as well as a timestamp). Doing so can, forexample, ensure consistent processing of multiple shards originatingfrom multiple producers.

At operation 215, the first shard (e.g., as generated at operation 205)is provided. For example, in the scenario depicted in FIG. 1A, producer110A and/or stream production engine 112 can provide a shard (e.g.,shard ‘S1’) into stream 140A (e.g., via a ‘push’ operation).

In certain implementations, such a shard (and an associated attribute)is provided (e.g., ‘pushed’) as an update (e.g., within a data stream).In certain implementations, such an update can be an atomic updateand/or a conditional update (e.g., within a data stream, such as anupdate that transforms shard 130D to shard 130D′, as shown in FIG. 1A).For example, an atomic update can include multiple updates, operations,etc., that are to be performed collectively (e.g., on repository 122).In doing so, either all of the updates, operations, etc., are to beperformed or the atomic update is rejected and none of the updates,operations, etc., are to be performed (e.g., in a scenario in whichcertain updates cannot be completed). By way of further example, theproviding of such a shard can be conditioned, for example, on it beingprovided to and/or received by a streaming system (e.g., server 120) forthe first time. Accordingly, upon determining, for example, that a shardhas already been provided/received (e.g., based on attributes/sequenceidentifier(s) of the received shard and/or other shards), the referencedupdate operation can be canceled. In other implementations, shard(s)that are received out of order can be handled in other ways, asdescribed herein.

By way of further illustration, in a scenario in which a producer (e.g.,device 110A) malfunctions or fails while providing messages, shards,etc., when such a producer resumes operation, attributes (e.g., sequenceidentifiers) of those messages/shards that have been previously providedto the stream can be used to determine that such shards/messages do notneed to be provided/pushed again. Doing so can enable shards, messages,records, etc., to be provided exactly once by a producer to a streamingsystem, and can eliminate the need for redundant records/shards, even inscenarios in which such producer fails.

At operation 220, an attribute such as a state attribute is requested(e.g., from the first shard). For example, in a scenario in whichproducer 110A malfunctions, fails, etc., when reinitializing, producer110A (and/or stream production engine 112) can request a state attribute(e.g., from stream 140A, server 120 and/or repository 122). Doing so canenable producer 110A to reinitialize and continue providing shards,messages, etc., that have not previously been ‘pushed’ (e.g., to stream140A, server 120, and/or repository 122), without providing or pushingredundant shard(s)/message(s) (which have already been pushed/received).

By way of further illustration, in certain implementations thereferenced attribute (being requested) can include or reflect a sequenceidentifier (e.g., from the first shard). For example, in a scenario inwhich producer 110A malfunctions, fails, etc., when reinitializing,producer 110A (and/or stream production engine 112) can request asequence identifier (e.g., from stream 140A, server 120 and/orrepository 122). Doing so can enable producer 110A to reinitialize andcontinue providing shards, messages, etc., that have not previously been‘pushed’ (e.g., to stream 140A, server 120, and/or repository 122),without providing or pushing redundant shard(s)/message(s) (which havealready been pushed/received).

At operation 225, an attribute such as a state attribute is received(e.g., in response to the request at operation 220). In certainimplementations, such a state attribute can be received by producer 110Aand/or stream production engine 112, as shown in FIG. 1A. Additionally,in certain implementations such an attribute can be received in responseto a request (e.g., the request provided at operation 220). As describedherein, such an attribute can reflect a state of the producer when suchshard, etc., was provided/pushed.

By way of illustration, in a scenario in which producer 110A (as shownin FIG. 1A) fails or malfunctions after pushing shard ‘S1,’ uponreinitializing, the producer can request the state attribute 152Aassociated with such a shard (e.g., from stream 140A and/or server 120).Upon receiving the associated state attribute (here, ‘X1’), producer110A can determine its state (e.g., at the time the shard, etc., waspushed) and can thus reinitialize operation to such a state (andcontinue pushing subsequent shard(s)). It can be appreciated that such aconfiguration can enable the producer to operate in a stateless manner.

By way of further illustration, in certain implementations a sequenceidentifier can be received. In certain implementations, such a sequenceidentifier can be received by producer 110A and/or stream productionengine 112, as shown in FIG. 1A. Additionally, in certainimplementations such a sequence identifier can be received in responseto a request (e.g., the request provided at operation 220). As describedherein, such a sequence identifier can reflect the relative position ofa shard (and/or message(s)) within a sequence.

By way of illustration, in a scenario in which producer 110A (as shownin FIG. 1A) fails or malfunctions after pushing shard ‘S1,’ uponreinitializing, the producer can request the sequence identifierassociated with such a shard (e.g., from stream 140A and/or server 120).Upon receiving the associated sequence identifier, producer 110A candetermine that shard ‘S1’ has been successfully pushed, and canreinitialize operation to continue pushing subsequent shard(s). Doing socan enable such shards, messages, records, etc., to be provided andprocessed ‘exactly once,’ without needing redundant operations and/ormultiple copies to ensure all messages are provided/processed.

At operation 230, a second shard is provided. In certainimplementations, such a shard can be provided within a data stream basedon the received state attribute(s) (e.g., as received at operation 225).By way of illustration, in a scenario in which producer 110A (as shownin FIG. 1A) fails or malfunctions after pushing shard ‘S1,’ uponreinitializing, the producer can request (e.g., at operation 220) thestate attribute 152A (e.g., a sequence identifier and/or anotherattribute) associated with such a shard (e.g., from stream 140A and/orserver 120). Upon receiving (e.g., at operation 225) the associatedstate attribute (here, ‘X1’), producer 110A can determine that shard‘S1’ has been successfully pushed, and can reinitialize operation tocontinue pushing other/subsequent shard(s) (e.g., shard ‘S2’). Doing socan enable such shards, messages, records, etc., to be provided andprocessed ‘exactly once,’ without needing redundant operations and/ormultiple copies to ensure all messages are provided/processed.

By way of further example, the referenced second shard can be providedwithin a data stream based on a received attribute (e.g., a stateattribute, such as is received at operation 225). By way ofillustration, in a scenario in which producer 110A (as shown in FIG. 1A)fails or malfunctions after pushing shard ‘S1,’ upon reinitializing, theproducer can request the state attribute 152A associated with such ashard (e.g., from stream 140A and/or server 120). Upon receiving theassociated state attribute (here, ‘X1’), producer 110A can determine itsstate (e.g., at the time the shard, etc., was pushed) and canreinitialize operation to such a state (and continue pushingother/subsequent shard(s)). As described herein, such a configurationcan enable the producer to operate in a stateless manner.

At operation 235, an operation of a message production source can beadjusted. That is, as described herein, the referenced shard(s) can beassociated with various attribute(s) (e.g., attribute 152B as shown inFIG. 1A). As also described herein, in certain implementations, producer110A can request and/or receive certain attributes (based upon which theproducer can, for example, determine which shards have/have not beenprovided within a stream). Accordingly, in certain implementations, suchattribute(s) can also be used to control or adjust operation of theproducer. For example, in certain implementations server 120 (e.g., astreaming system/service) can associate, assign, or update certainattribute(s) with respect to a shard (e.g., as stored in repository122). Upon receiving a request from the producer (e.g., as describedabove at operation 220), such control attribute(s) can be provided tothe producer. Upon receiving such attribute(s) (e.g., as described atoperation 225), the producer can adjust its operation accordingly.Additionally, in certain implementations, such operation(s) can beadjusted based on a received attribute (e.g., as received at operation225).

By way of illustration, in one scenario such an attribute 152B canreflect whether the producer is (or is not) to remain active. Such anattribute can be dictated or provided by another entity (e.g., streamingsystem 120 and/or another source). Accordingly, upon receiving a shard(e.g., shard ‘S1’ as shown in FIG.1A) and storing/maintaining such ashard (e.g., within repository 122), server 120 can associate attribute152B to the shard which can reflect, for example, that producer 110A isto be disabled. Upon receiving a request (e.g., from producer 110A) forsuch attribute(s), the ‘disable’ attribute can be returned (and receivedby producer 110A). Producer 110A can then adjust its operation (here,disabling itself from providing subsequent shards, messages, etc.)and/or perform corresponding operations. In doing so, push/pulloperations initiated by the described producer can be used to enableadditional functionality.

Additionally, in certain implementations, the referenced attribute 152B(which can be used to adjust/control operation of the producer) caninclude/reflect a token. Such a token may be assigned (e.g., to a shard,message, etc.) based on a processing capacity of a streaming system. Incertain implementations, such a token can be assigned by the streamingsystem 120 and/or by a consumer (e.g., consumer 110C) of the stream. Forexample, the referenced tokens can be used to implement flow controloperations which can, for example, adjust operation of the producer(e.g., in scenarios in which shards, messages, etc., are being providedtoo quickly).

By way of illustration, system 120 (and/or a consumer) can assign atoken to a shard, e.g., as attribute 152B as shown in FIG. 1A. Thesystem can be configured to assign a certain number of tokens (e.g.,1000 tokens per second to a stream originating from a particularproducer) and may be further configured to only store/maintain thoseshards being assigned a token (e.g., in repository 122). The system 120can also be configured to adjust (e.g., increase or decrease) the numberof tokens (e.g., in scenarios in which it may be advantageous for systemand/or consumer(s) to increase/decrease the rate at which messages,shards, etc., are being received from the stream). While those shards,messages, etc., can be stored/maintained (e.g., in repository 122),those shards, messages, etc. not assigned a token may not bestored/maintained (until they are assigned a token). In doing so, theflow of shards, messages, etc., from the producer can be furthercontrolled using push/pull operations initiated by the producer (withouta separate control channel to control operation).

FIG. 2B is a flow chart illustrating a method 240, according to anexample embodiment, for stateless stream handling and resharding. Themethod is performed by processing logic that can comprise hardware(circuitry, dedicated logic, etc.), software (such as is run on acomputing device such as those described herein), or a combination ofboth. In one implementation, the method 240 is performed by one or moreelements depicted and/or described in relation to FIG. 1A (including butnot limited to server 120 and/or stream management engine 124), while insome other implementations, the one or more blocks of FIG. 2B can beperformed by another machine or machines.

At operation 250, a first shard is received, e.g., within a firststream. In certain implementations, such a shard can include one or moremessages, events, records, etc., as described herein (e.g., shard ‘S1’including include messages 132, as shown in FIG. 1A). Additionally, incertain implementations, such a shard can include or reflect a stateattribute. Such a state attribute 152A can reflect, for example, aspectsof the state of the producer, such as the number of messages beingproduced by the producer (e.g., per second), aspects of datatransformation being performed by the producer, and/or other state(s)reflecting the status or operation(s) of the producer.

In other implementations, such a state attribute can reflect theposition of the associated shard (and/or message(s)) within a sequence.By way of illustration, such a state attribute can include or reflect atime/date stamp (reflecting, for example, the time/date the associatedshard, message(s), etc., was/were received, created, and/or provided).In doing so, the relative position of a certain shard can be determined.

At operation 255, a message is identified (e.g., within the shardreceived at operation 250). In certain implementations such a message(or messages) can be one that is inconsistent with a state attributeassociated with the shard within which the message was received. Forexample, such a message can be identified as being received out ofsequence with one or more other messages within the first shard. Forexample, in the scenario depicted in FIG. 1A, message ‘M2’ can bedetermined to have been received out of order (e.g., with respect toother message(s) within the shard). In certain implementations, such adetermination can be computed based on the state attribute 152Aassociated with the shard. For example, the referenced state attributecan reflect that the shard includes messages received/provided during acertain period of time, while the message (‘M2’) may reflect a messagefrom another period of time.

At operation 260, the message (e.g., as identified at operation 255) isassociated as an attribute of the first shard. For example, an attribute152B of the shard (e.g., shard ‘S1’) can be populated with the content,data, etc. of such a message (which has been determined not to belongwithin the sequence of other message(s) within the shard. Such anattribute can function as a queue for such message(s) (e.g., thosereceived out-of-sequence), as described herein.

At operation 265, a second shard is received. In certainimplementations, such a shard can include or reflect another stateattribute. Additionally, in certain implementations such a second shardcan be received within the first stream (e.g., the stream within whichthe first shard was received at operation 250) and/or within a secondstream (which may originate from another producer). By way ofillustration, additional shard(s) (e.g., shard ‘S2,’ ‘S3,’ etc.) can bereceived, e.g., from the same producer and/or from other producer(s).

At operation 270, a position of the message within the second shard isdetermined. In certain implementations, such position can be determined(e.g., based on the second state attribute). For example, upon receivingother shard(s), it can be further determined whether the identifiedmessage(s) (which were received out-of-order within the first shard) arecorrectly positioned within the other shard(s). As described herein, thecorrect position of such message(s) can be determined based on therespective state attribute(s) of the received shard(s).

At operation 275, the message can be inserted into the second shard(e.g., based on the determining). For example, in the scenario depictedin FIG. 1A, upon determining that the correct position of message ‘M2’is within shard ‘S2,’ stream management engine 124 can insert themessage into the appropriate shard. It can be appreciated that doing socan, for example, enable multiple streams to be processed together, evenin scenarios in which they may not be perfectly aligned.

Additionally, in certain implementations the described technologies canbe configured to process the described streams, shards, messages, etc.,in order to identify gaps within the referenced data/content. Forexample, in scenarios in which data pushed into a shard is expected tobe sorted and/or identified using various record identifiers, etc. gapswithin such data (reflecting, for example, missing records) can beidentified and/or recorded/saved. In certain implementations, such gapscan be identified based on the described state attribute(s) which canreflect the position of a shard, message, etc., e.g., within a sequence.Upon identifying such a gap, various alerts can be initiated/provided(e.g., to attempt to locate the missing records, to highlight such adeficiency to an administrator, etc.).

FIG. 3A is a flow chart illustrating a method 300, according to anexample embodiment, for stateless stream handling and resharding. Themethod is performed by processing logic that can comprise hardware(circuitry, dedicated logic, etc.), software (such as is run on acomputing device such as those described herein), or a combination ofboth. In one implementation, the method 300 is performed by one or moreelements depicted and/or described in relation to FIG. 1A (including butnot limited to device 110C and/or stream consumption engine 114), whilein some other implementations, the one or more blocks of FIG. 3 can beperformed by another machine or machines.

At operation 305, a first shard is received (e.g., from server 120and/or producer 110A). In certain implementations, such a shard caninclude one or more messages, events, records, etc., as describedherein. For example, as shown in FIG. 1B, shard ‘S4’ (130E) can bereceived from server 120. As noted, in certain implementations such ashard may originate at producer 110A. As shown in FIG. 1B, shard 130Ecan include messages 132 (e.g., messages ‘M4,’ ‘M5,’ ‘M6,’ etc.).

At operation 310, the first shard (e.g., as received at operation 305)is associated with an attribute. In certain implementations, such anattribute can be a state attribute 152C, such as state attribute(“STATE”) ‘X2,’ as shown in FIG. 1B. Such a state attribute can reflect,for example, various aspects of the state of the consumer. Examples ofsuch a state can include but are not limited to a quantity or valuecorresponding to the number of messages being pulled, received and/orprocessed by the consumer (e.g., per second), various aspects of datatransformation being performed by the consumer, and/or other state(s)reflecting the status or operation(s) of the consumer.

In certain scenarios, storing/associating a state attribute with a shardcan enable a consumer that fails or malfunctions to be reinitialized andcontinue pulling, requesting and/or processing shards, messages,records, etc., within a stream. For example, as described herein, in ascenario in which a consumer fails and is reinitialized, the consumercan request (e.g., from server 120 and/or repository 122) the stateattribute (e.g., from a shard associated with the same consumer andprovided to/received by server 120). In response, the consumer canreceive a state attribute that reflects, for example, the state of theconsumer (e.g., when the most recently received shard was provided orprocessed). The consumer can then reinitialize and continue pulling orprocessing shards, messages, etc. based on such received state (ratherthan, for example, pulling or processing redundant copies ofshards/records that have already been received, processed and/orprovided to the streaming system). It can be appreciated that such aconfiguration can enable the consumer to operate in a stateless manner.

Additionally, as shown in FIG. 1B, in certain implementations thedescribed technologies can further assign or associate additionalattributes 152D to the referenced shards, messages, etc. Such attributescan, for example, enable various entities, services, systems, etc.(e.g., server 120) to collect, monitor, and/or generate various metrics,statistics, etc., that reelect aspects of the operation of a consumer.By way of illustration, such attributes (which can be associated to ashard, e.g., by the consumer from which it is received) can reflect thenumber of messages, events, etc. pulled by the consumer, number ofrecords updated, messages since a last pull operation, various latenciesassociated with operations of the consumer (e.g., pull latency), etc. Indoing so, the streaming system can monitor the operation of variousconsumer(s), and can further adjust various other operations based onthe referenced metrics, statistics, etc., as described herein.

Additionally, in certain implementations such a state attribute canreflect an importance and/or location of one or more of the messages(e.g., within the associated shard/stream). By way of illustration, in ascenario in which messages, data, etc., being received, pulled, and/orprocessed by the consumer have a structured format, the describedtechnologies (e.g., stream consumption engine 114) can enable variousoperations/transformations. For example, an attribute (e.g., attribute152D as shown in FIG. 1B) and/or other such property of a shard (e.g.,shard ‘S4’) can be assigned/updated based on message(s), data, etc.,within the shard. By way of illustration, such attribute(s),propertie(s), etc. can be used for statistics (e.g., reflecting messageproperties such as message types), alerts (e.g., based on content of amessage within the stream), location markers (e.g., reflecting locationof certain messages within a shard), etc.

By way of further illustration, it can be appreciated that certainmessages received, pulled, and/or processed by a consumer may be ofparticular significance, importance, etc. (e.g., messages containingcontent that may necessitate immediate action). Accordingly, it can beadvantageous to configure the described technologies to enable suchmessage(s) to be easily accessed, identified, etc. In certainimplementations, when receiving, pulling, and/or processing a shard thatincludes message(s) of particular importance (e.g., messages containingcertain types of content), an attribute 152D or other such property canbe associated with the shard, reflecting that it contains an importantmessage. Upon receiving such a shard (with the referencedattribute/property), streaming system 120 can prioritize the processing,analysis, etc. of such a shard/message (and/or perform otheroperations). By way of further example, the referenced attribute 152D orproperty can reflect the location (e.g., within the shard) of suchimportant, significant, etc. message(s). In doing so, thestate/attributes of the shard can reflect content within its messagesand can further enable operations to be performed on such messages(e.g., in a prioritized manner).

Additionally, in certain implementations, the referenced attribute 152D(which can be used to adjust/control operation of a producer) caninclude/reflect a token, such as may be assigned based on a processingcapacity of a streaming system. Such a token may be assigned (e.g., to ashard, message, etc.) based on a processing capacity of a streamingsystem. In certain implementations, such a token can be assigned by thestreaming system 120 and/or by a consumer (e.g., consumer 110C) of thestream. For example, the referenced tokens can be used to implement flowcontrol operations which can, for example, adjust operation of theproducer (e.g., in scenarios in which shards, messages, etc., are beingprovided too quickly). Further aspects of the referenced flow controloperations are described below, e.g., at operation 350.

In other implementations, the referenced attribute can include orreflect an identifier such as a sequence identifier. Such a sequenceidentifier can reflect the position of the associated shard (and/ormessage(s)) within a sequence. By way of illustration, a time/date stamp(reflecting, for example, the time/date the associated shard,message(s), etc., was/were received and/or processed) can be used as asequence identifier. In doing so, the relative position of a certainshard can be determined. For example, a sequence identifier associatedwith shard ‘S5’ (as shown in FIG. 1B) can reflect that such a shard wasreceived and/or processed after shard ‘S4’ and before shard ‘S6’ (withinstream 140C).

It should be noted that in scenarios in which multiple consumers arepresent (e.g., as shown in FIG. 1B), the referenced sequenceidentifier(s) can further include a field, property, etc., that reflectsthe consumer that pulled, processed, etc. the associatedshard/message(s). Accordingly, in the scenario depicted in FIG. 1B,shard(s) received/processed by consumer 110C can be associated with anattribute reflecting the identity of the consumer and a timestamp, whileshard(s) received/processed by consumer 110D can be associated withcorresponding attribute(s) also reflecting the identity of that producer(as well as a timestamp). Doing so can, for example, ensure consistentprocessing of multiple shards across multiple consumers.

At operation 315, the first shard (e.g., as received at operation 305)is provided. For example, in the scenario depicted in FIG. 1B, consumer110C and/or stream consumption engine 114 can provide a shard (e.g.,shard ‘S4’) into stream 140C (e.g., via a ‘push’ operation).

In certain implementations, such a shard (and an associated stateattribute) is provided (e.g., ‘pushed’) as an update (e.g., within adata stream). In certain implementations, such an update can be anatomic update and/or a conditional update (e.g., within a data stream,such as an update that transforms shard 130H to shard 130H′, as shown inFIG. 1B). For example, an atomic update can include multiple updates,operations, etc., that are to be performed collectively. In doing so,either all of the updates, operations, etc., are to be performed or theatomic update is rejected and none of the updates, operations, etc., areto be performed (e.g., in a scenario in which certain updates cannot becompleted). By way of further example, the providing of such a shard canbe conditioned, for example, on it being provided to and/or received bya streaming system (e.g., server 120) for the first time. Accordingly,upon determining, for example, that other shards have already beenprovided/received (e.g., based on attributes/sequence identifier(s) ofthe received shard and/or other shards), the referenced update operationcan be canceled. In other implementations, shard(s) that are receivedout of order can be handled in other ways, as described herein.

By way of further illustration, in a scenario in which a consumer (e.g.,device 110C) malfunctions or fails while providing messages, shards,etc., when such a consumer resumes operation, the attribute(s) (e.g.,sequence identifiers) of those messages/shards that have been previouslyprovided to the stream can be used to determine that suchshards/messages do not need to be provided/pushed again. Doing so canenable shards, messages, records, etc., to be provided exactly once by aconsumer to a streaming system, and can eliminate the need for redundantrecords/shards, even in scenarios in which such consumer fails.

At operation 320, an attribute such as a state attribute is requested(e.g., from the first shard). For example, in a scenario in whichconsumer 110C malfunctions, fails, etc., when reinitializing, consumer110C (and/or stream consumption engine 114) can request a stateattribute (e.g., from stream 140C, server 120 and/or repository 122).Doing so can enable consumer 110C to reinitialize and continue pullingor processing shards, messages, etc., that have not previously beenhandled (e.g., to stream 140C, server 120, and/or repository 122),without pulling or processing redundant shard(s)/message(s) (which havealready been handled).

By way of further illustration, in certain implementations thereferenced attribute (being requested) can include or reflect a sequenceidentifier (e.g., from the first shard). For example, in a scenario inwhich consumer 110C malfunctions, fails, etc., when reinitializing,consumer 110C (and/or stream consumption engine 114) can request asequence identifier (e.g., from stream 140C, server 120 and/orrepository 122). Doing so can enable consumer 110C to reinitialize andcontinue providing shards, messages, etc., that have not previously been‘pushed’ (e.g., to stream 140C, server 120, and/or repository 122),without providing or pushing redundant shard(s)/message(s) (which havealready been pushed/received).

At operation 325, an attribute such as a state attribute is received(e.g., in response to the request at operation 320). In certainimplementations, such a state attribute can be received by consumer 110Cand/or stream consumption engine 114, as shown in FIG. 1B. Additionally,in certain implementations such an attribute can be received in responseto a request (e.g., the request provided at operation 320). As describedherein, such an attribute can reflect a state of the consumer when suchshard, etc., was pulled/processed.

By way of illustration, in a scenario in which consumer 110C (as shownin FIG. 1B) fails or malfunctions after pulling or processing shard‘S4,’ upon reinitializing, the consumer can request the state attribute152C associated with such a shard (e.g., from stream 140C and/or server120). Upon receiving the associated state attribute (here, ‘X2’),consumer 110C can determine its state (e.g., at the time the shard,etc., was pulled or processed) and can thus reinitialize operation tosuch a state (and continue pulling/processing subsequent shard(s)). Itcan be appreciated that such a configuration can enable the consumer tooperate in a stateless manner.

By way of further illustration, in certain implementations a sequenceidentifier can be received. In certain implementations, such a sequenceidentifier can be received by consumer 110C and/or stream consumptionengine 114, as shown in FIG. 1C. Additionally, in certainimplementations such a sequence identifier can be received in responseto a request (e.g., the request provided at operation 320). As describedherein, such a sequence identifier can reflect the relative position ofa shard (and/or message(s)) within a sequence.

By way of illustration, in a scenario in which consumer 110C (as shownin FIG. 1B) fails or malfunctions after pushing shard ‘S4,’ uponreinitializing, the consumer can request the sequence identifierassociated with such a shard (e.g., from stream 140C and/or server 120).Upon receiving the associated sequence identifier, consumer 110C candetermine that shard ‘S4’ has been successfully pushed, and can thusreinitialize operation to continue pushing, processing, etc. subsequentshard(s). Doing so can enable such shards, messages, records, etc., tobe pulled and processed ‘exactly once,’ without needing redundantoperations and/or multiple copies to ensure all messages arepulled/processed.

At operation 330, a second shard is provided. In certainimplementations, such a shard can be provided within a data stream basedon the received state attribute(s) (e.g., as received at operation 325).By way of illustration, in a scenario in which consumer 110C (as shownin FIG. 1B) fails or malfunctions after pulling or processing shard‘S4,’ upon reinitializing, the consumer can request (e.g., at operation320) the state attribute 152C associated with such a shard (e.g., fromstream 140C and/or server 120). Upon receiving (e.g., at operation 325)the associated state attribute (here, ‘X2’), consumer 110C can determinethat shard ‘S4’ has been successfully pulled/processed, and canreinitialize operation to continue pulling or processing subsequentshard(s) (e.g., shard ‘S5’). Doing so can enable such shards, messages,records, etc., to be pulled and processed ‘exactly once,’ withoutneeding redundant operations and/or multiple copies to ensure allmessages are handled.

By way of further example, the referenced second shard can be providedwithin a data stream based on a received attribute (e.g., a stateattribute, such as is received at operation 325). By way ofillustration, in a scenario in which consumer 110C (as shown in FIG. 1B)fails or malfunctions after pulling or processing shard ‘S4,’ uponreinitializing, the consumer can request the state attribute 152Cassociated with such a shard (e.g., from stream 140C and/or server 120).Upon receiving the associated state attribute (here, ‘X2’), consumer110C can determine its state (e.g., at the time the shard, etc., waspulled/processed) and can reinitialize operation to such a state (andcontinue pulling/processing subsequent shard(s)). As described herein,such a configuration can enable the consumer to operate in a statelessmanner.

At operation 335, an adjustment of an operation of a message productionsource can be initiated. That is, as described herein, the referencedshard(s) can be associated with various attribute(s) (e.g., attribute152D as shown in FIG. 1B). As also described herein, in certainimplementations, producer 110A can request and/or receive certainattributes (based upon which the producer can, for example, determinewhich shards have/have not been provided within a stream). Accordingly,in certain implementations, such attribute(s) can also be used tocontrol or adjust operation of the producer. For example, in certainimplementations server 120 (e.g., a streaming system/service) and/orconsumer 110C can associate, assign, or update certain attribute(s) withrespect to a shard (e.g., as stored in repository 122). Upon receiving arequest from the producer (e.g., as described above at operation 220),such control attribute(s) can be provided to the producer. Uponreceiving such attribute(s) (e.g., as described at operation 225), theproducer can adjust its operation accordingly. Additionally, in certainimplementations, the adjustment of such operation(s) by producer 110Acan be initiated by consumer 110C via updates to the described stateattribute(s).

By way of illustration, in one scenario such an attribute 152D canreflect whether the producer is (or is not) to remain active. Such anattribute can be dictated or provided by another entity (e.g., streamingsystem 120, consumer 110C, and/or another source). Accordingly, uponreceiving a shard (e.g., shard ‘S4’ as shown in FIG.1B) andstoring/maintaining such a shard (e.g., within repository 122), server120 can associate attribute 152D to the shard which can reflect, forexample, that producer 110A is to be disabled. Upon receiving a request(e.g., from producer 110A) for such attribute(s), the ‘disable’attribute can be returned (and received by producer 110A). Producer 110Acan then adjust its operation (here, disabling itself from providingsubsequent shards, messages, etc.) and/or perform correspondingoperations. In doing so attributes/identifiers originating from aconsumer can be used to initiate operations by the described producer(which instructions can be transmitted via push/pull operations of theproducer), thereby enabling additional functionality.

Additionally, in certain implementations, the referenced attribute 152D(which can be used to adjust/control operation of the producer) caninclude/reflect a token. Such a token may be assigned (e.g., to a shard,message, etc.) based on a processing capacity of a streaming systemand/or a consumer. In certain implementations, such a token can beassigned by the streaming system 120 and/or by a consumer (e.g.,consumer 110C) of the stream. For example, the referenced tokens can beused to implement flow control operations which can, for example, adjustoperation of the producer (e.g., in scenarios in which shards, messages,etc., are being provided too quickly).

By way of illustration, system 120 (and/or a consumer) can assign atoken to a shard, e.g., as attribute 152D as shown in FIG. 1B. Thesystem can be configured to assign a certain number of tokens (e.g.,1000 tokens per second to a stream originating from a particularproducer) and may be further configured to only store/maintain thoseshards being assigned a token (e.g., in repository 122). The system 120can also be configured to adjust (e.g., increase or decrease) the numberof tokens (e.g., in scenarios in which it may be advantageous for systemand/or consumer(s) to increase/decrease the rate at which messages,shards, etc., are being received from the stream). While those shards,messages, etc., can be stored/maintained (e.g., in repository 122),those shards, messages, etc. not assigned a token may not bestored/maintained (until they are assigned a token). In doing so, theflow of shards, messages, etc., from the producer can be furthercontrolled (e.g., by a consumer) using push/pull operations initiated bythe producer (without a separate control channel to control operation).

FIG. 5A is a flow chart illustrating a method 510, according to anexample embodiment, for stateless stream handling and resharding. Themethod is performed by processing logic that can comprise hardware(circuitry, dedicated logic, etc.), software (such as is run on acomputing device such as those described herein), or a combination ofboth. In one implementation, the method 510 is performed by one or moreelements depicted and/or described in relation to FIG. 4A (including butnot limited to device 410A and/or stream production engine 112), whilein some other implementations, the one or more blocks of FIG. 5A can beperformed by another machine or machines.

At operation 512, a first shard is generated. In certainimplementations, such a shard can include or incorporate variousmessages, events, records, etc., as described herein. For example, asshown in FIG. 4A, shard 430A (‘S1’) can be generated by producer 410A.Such a shard can include messages 432 (‘M1’-‘M4’).

At operation 514, the first shard (e.g., the shard generated atoperation 512) can be associated with an attribute such as a shardversion attribute (e.g., attribute 458A, as shown in FIG. 4A). Such ashard version attribute can reflect, for example, a number or value thatcorresponds to the version of the shard (e.g., as generated by producer410A). That is, it can be appreciated that while messages, data,records, etc., within a stream can be divided up into shards, the size(and/or other aspects) of such shards may be suboptimal (e.g., inscenarios in which the streaming system and/or consumers cannot processsuch shards efficiently/optimally). Accordingly, as described herein,the referenced producer can be configured to re-shard the describedshard(s), in order to enable previously pushed records to be pushedwithin shards that may provide better results, efficiency, etc., whenhandled by the described technologies. In certain implementations, inorder to ensure consistency, such shards can be assigned a versionnumber or value (the described shard version attribute) to ensure thatup-to-date or most current shard(s) are processed (in lieu of previouslypushed shards which have since been re-sharded).

Additionally, as described herein, the disclosed technologies can enablevarious operations, such as atomic updates and conditional updates, tobe performed with respect to shard(s)/stream(s), e.g., based on thereferenced shard version attribute(s). For example, a push operation canbe generated/provided with a condition that reflects a particular shardversion attribute. Accordingly, in a scenario in which the shard versionchanges, such a push operation can be rejected (as described herein).

At operation 516, the first shard (e.g., as generated at operation 512)and the first shard version attribute (e.g., as associated at operation514) are provided or pushed e.g., as an update within data stream 440A(e.g., to system 120 and/or consumer 410B, as shown in FIG. 4A). Forexample, as shown in FIG. 4A, shard 430A can be pushed or provided byproducer 410A. As noted, such an update can be, for example, an atomicupdate that includes multiple updates, operations, etc., that are to beperformed collectively. In doing so, either all of the updates,operations, etc., are to be performed or the atomic update is rejectedand none of the updates, operations, etc., are to be performed (e.g., ina scenario in which certain updates cannot be completed).

At operation 518, a state attribute is received (e.g., from system 120and/or consumer 410B). In certain implementations, such a stateattribute can reflect a processing capacity of a streaming system and/ora consumer. By way of illustration, such an attribute can reflectbandwidth, resources, etc., of various available producers.

At operation 520, the first shard (e.g., as generated at operation 512)is resharded, e.g., into a second shard, third shard, etc. In certainimplementations, such resharding can be performed based on the firstshard. For example, a second shard (e.g., shard 430B as shown in FIG.4A) can be generated. Such a second shard can include message(s) (e.g.,messages ‘M1’ and ‘M2’) originating from the first shard 430A.

Additionally, in certain implementations the referenced resharding canbe performed or initiated based on the received first state attribute(e.g., at operation 518). For example, such a state attribute canreflect that the system 120 and/or consumer 410B may be overloaded orotherwise incapable of efficiently handling/processing the shardsoriginating from producer 410A. In response, the producer can reshardpreviously pushed shards (e.g., shard 430A as shown in FIG. 4A), e.g.,in to new shards (430B and 430C, which can contain fewer messages pershard, as shown). In doing so, such shards (even those that have alreadybeen pushed) can be updated in a manner that enables them to be handled,processed, etc., more efficiently (e.g., by multiple consumers).

At operation 522, a third shard is generated (e.g., in accordance withthe resharding at operation 520). For example, as shown in FIG. 4A,shard 430C can be generated.

At operation 524, the second shard (e.g., as resharded/generated atoperation 520) is associated with a second shard version attribute. Asdescribed herein, such a shard version attribute can reflect, forexample, a number or value that corresponds to the version of the shard(e.g., as generated by producer 410A). For example, while shard 430A isassociated with a shard version attribute 458A (‘VERSION: 1’), shard430B is associated with a shard version attribute 458B (‘VERSION: 2,’reflecting that it is a newer, updated version of shard 430A).

Additionally, in certain implementations, in scenarios in which a shardis resharded (e.g., shard 430A as shown in FIG. 4A), an attributereflecting a point or location within a shard or stream that correspondsto the resharding operation can be associated with the referenced shard.In certain implementations, such an attribute can be persisted in anatomic manner. For example, attribute 458N as shown in FIG. 4A can beassociated with shard 430A (which is being resharded, as describedherein). Such an attribute can reflect a point or location within theshard/stream that corresponds to the resharding operation (e.g., priorto/at the pushing and/or processing of message ‘M1’). Maintaining such apoint/location as an attribute of the shard/stream can be advantageous,for example, in enabling identification of the point at which thereferenced resharding occurred. Doing so can enable multiple consumersto synchronize their operations, e.g., to ensure that messages from thereferenced shard/stream are only processed once.

At operation 526, the second shard and the second shard versionattribute are provided or pushed e.g., as an update (e.g., an atomicupdate or a conditional update) within data stream 440A (e.g., to system120 and/or consumer 410B, as shown in FIG. 4A). For example, as shown inFIG. 4A, shard 430B can be pushed or provided by producer 410A. Incertain implementations, such a second shard can be provided/pushed asan update within the data stream in lieu of another update (e.g., theupdate provided at operation 516). For example, such a push operationcan include conditions that reflect particular shard version attributes.Accordingly, in a scenario in which the shard version changes, pushoperations that do not correspond to the updated shard version attributecan be rejected. In doing so, updated shard(s) can be provided/pushed,thereby enabling more efficient operation of the system and/orconsumers. By way of further example, such an update can be, forexample, an atomic update that includes multiple updates, operations,etc., that are to be performed collectively. In doing so, either all ofthe updates, operations, etc., are to be performed or the atomic updateis rejected and none of the updates, operations, etc., are to beperformed (e.g., in a scenario in which certain updates cannot becompleted).

FIG. 5B is a flow chart illustrating a method 530, according to anexample embodiment, for stateless stream handling and resharding. Themethod is performed by processing logic that can comprise hardware(circuitry, dedicated logic, etc.), software (such as is run on acomputing device such as those described herein), or a combination ofboth. In one implementation, the method 530 is performed by one or moreelements depicted and/or described in relation to FIG. 4A (including butnot limited to server 120 and/or stream management engine 124), while insome other implementations, the one or more blocks of FIG. 5B can beperformed by another machine or machines.

At operation 532, a first shard is received, e.g., from a device (e.g.,producer 410A as shown in FIG. 4A and described herein. In certainimplementations, such a shard (e.g., shard 430A) can include variousmessage(s) 432 and attribute(s) such as a shard version attribute 458A.As described herein, such a shard version attribute can reflect, forexample, a number or value that corresponds to the version of the shard.

At operation 534, a current shard version is requested, e.g., from thedevice (e.g., producer 410A as shown in FIG. 4A). For example, asdescribed herein, though producer 410A may have pushed/provided shard430A within stream 440A, the producer may have subsequently re-shardedthe shard (e.g., by generating shards 430B, 430C, etc.). Accordingly,prior to processing, handling, etc. operation(s) associated with shard430A, the current shard version can be requested (e.g., from producer410A). In doing so, it can be determined/confirmed (e.g., based on acomparison of the current shard version provided by the producer and theshard version attribute of the received shard) whether the receivedshard is still the current version, or whether subsequent shard versionshave been generated (and should be handled in lieu of the previousshard).

At operation 536, an operation, transformation, etc. is performed withrespect to the first shard (e.g., the shard received at operation 532).In certain implementations, such an operation (e.g., providing the firstshard to a consumer, etc.), can be performed based on a determinationthat the current shard version (e.g., as received from producer 410Aand/or identified within stream 440A) is consistent with the first shardversion attribute (e.g., the shard attribute associated with the shardas received at operation 532).

At operation 538, performance of the operation with respect to the firstshard can be canceled. In certain implementations, such operation can becanceled based on a determination that the current shard version (e.g.,as requested/received at operation 534) is not consistent with the firstshard version attribute (e.g., the shard attribute associated with theshard as received at operation 532). For example, in the scenariodepicted in FIG. 4A, system 120 can determine (e.g., based on an input,attribute, etc., originating from producer 410A) that the current shardversion (e.g., as reflected in shards 430B, 430C, etc.) is version ‘2.’Accordingly, operations (e.g., processing, handling, etc.) associatedwith shard 430A (which corresponds to shard version ‘1’) can be canceled(as such a shard has since been re-sharded, as described herein). Indoing so, those shards that are up-to-date/current can be processedwhile those that are not current can be avoided, dropped, canceled, etc.It should be understood that the messages within the referenced shards(‘M1’-‘M4’) are processed ‘exactly once,’ without necessitating multipleredundant processing instances for the same messages.

Additionally, as noted above, in certain implementations, a shard thatis resharded (e.g., shard 430A as shown in FIG. 4A), can include/beassociated with an attribute reflecting a point or location within theshard or stream that corresponds to the resharding operation. Forexample, attribute 458N can be associated with shard 430A (which isbeing resharded), reflecting the point/location within the shard/streamthat corresponds to the resharding operation (e.g., prior to/at thepushing and/or processing of message ‘M1’). Accordingly, in certainimplementations, operations associated with/directed to such ashard/stream can be performed up to the point/location reflected in thereferenced attribute. Operations associated with points/locations withinsuch a shard/stream that are subsequent to the referenced point/location(e.g., within a sequence) can be canceled or rejected (as suchoperations are to be performed with respect to the subsequent version(s)of the shard, as described herein). Doing so can enable multipleconsumers to synchronize their operations, e.g., to ensure that messagesfrom the referenced shard/stream are only processed once.

At operation 540, a second shard is received, e.g., from the referencedproducer (e.g., producer 410A as shown in FIG. 4A). In certainimplementations, such a second shard (e.g., shard 430B) can include asecond shard version attribute (e.g., attribute 458B, as shown in FIG.4A and described herein).

At operation 542, an operation is performed with respect to the secondshard (e.g., shard 430B as received at operation 540). In certainimplementations, such an operation (e.g., an update or other suchprocessing operation) can be performed with respect to the second shard(e.g., shard 430B as received at operation 540) in lieu of performingsuch an operation with respect to the first shard (e.g., shard 430A asreceived at operation 532). In certain implementations, such anoperation can be performed (e.g., with respect to shard 430B) based on adetermination that the current shard version (e.g., of the producer410A) is consistent with the second shard version attribute (e.g.,‘VERSION: 2’).

FIG. 5C is a flow chart illustrating a method 550, according to anexample embodiment, for stateless stream handling and resharding. Themethod is performed by processing logic that can comprise hardware(circuitry, dedicated logic, etc.), software (such as is run on acomputing device such as those described herein), or a combination ofboth. In one implementation, the method 550 is performed by one or moreelements depicted and/or described in relation to FIG. 4B (including butnot limited to device 410B and/or stream consumption engine 114), whilein some other implementations, the one or more blocks of FIG. 5C can beperformed by another machine or machines.

At operation 552, a first shard is received (e.g., by consumer 410B asshown in FIG. 4B). In certain implementations, such a shard can includevarious message(s) and attribute(s) such as a shard version attribute.For example, as shown in FIG. 4B, shard 430D can be received by consumer410B. Such a shard 430D can include messages 432 (‘M5’-‘M8’) and shardversion attribute 458D (‘VERSION: 1’). As described herein, such a shardversion attribute can reflect, for example, a number or value thatcorresponds to the version of the shard 430D.

At operation 554, a current shard version is requested. In certainimplementations, such a current version can be requested from system 120and/or producer 410A (e.g., the producer from which the shardoriginated). For example, as described herein, though producer 410A mayhave pushed/provided shard 430D, the producer may have subsequentlyre-sharded the shard (e.g., by generating shards 430E, 430F, etc., asshown in FIG. 4B). Accordingly, prior to processing, handling, etc.operation(s) associated with shard 430D, the current shard version canbe requested. In doing so, it can be determined/confirmed whether thereceived shard 430D is still the current version, or whether subsequentshard versions have been generated (and should be handled in lieu of theprevious shard).

At operation 556, an operation, transformation, etc. is performed withrespect to the first shard (e.g., the shard received at operation 552).In certain implementations, such an operation (e.g., analyzing,processing, etc. the first shard), can be performed based on adetermination that the current shard version (e.g., as received fromproducer 410A or system 120 and/or identified within stream 440B) isconsistent with the first shard version attribute (e.g., the shardattribute associated with the shard as received at operation 552).

At operation 558, performance of the operation with respect to the firstshard can be canceled. In certain implementations, such operation can becanceled based on a determination that the current shard version (e.g.,as requested/received at operation 554) is not consistent with the firstshard version attribute (e.g., the shard attribute associated with theshard as received at operation 552). For example, in the scenariodepicted in FIG. 4B, it can be determined that the current shard version(e.g., as reflected in shards 430E, 430F, etc.) is version ‘2.’Accordingly, processing, handling, etc., of shard 430D (whichcorresponds to shard version ‘1’) can be canceled (as such a shard hassince been re-sharded, as described herein). In doing so, those shardsthat are up-to-date/current can be processed while those that are notcurrent can be avoided, dropped, canceled, etc. It should be understoodthat the messages within the referenced shards (‘M5’-‘M8’) are processed‘exactly once,’ without necessitating multiple redundant processinginstances for the same messages.

Additionally, as noted above, in certain implementations, a shard thatis resharded (e.g., shard 430D as shown in FIG. 4B), can include/beassociated with an attribute reflecting a point or location within theshard or stream that corresponds to the resharding operation. Forexample, attribute 458N can be associated with shard 430D (which isbeing resharded), reflecting the point/location within the shard/streamthat corresponds to the resharding operation (e.g., prior to/at thepushing and/or processing of message ‘M5’). Accordingly, in certainimplementations, operations associated with/directed to such ashard/stream can be performed up to the point/location reflected in thereferenced attribute. Operations associated with points/locations withinsuch a shard/stream that are subsequent to the referenced point/location(e.g., within a sequence) can be canceled or rejected (as suchoperations are to be performed with respect to the subsequent version(s)of the shard, as described herein). Doing so can enable multipleconsumers to synchronize their operations, e.g., to ensure that messagesfrom the referenced shard/stream are only processed once.

At operation 560, a resharding request is provided. In certainimplementations, such a request can be provided to system 120 and/orproducer 410A. Such a resharding request can be provided, for example,based on a processing capacity (and/or other aspects, resources, etc.)of the producer 410B (and/or other producers). By way of illustration,upon determining that the consumer cannot efficiently or optimallyprocess shards with four (or more) messages, a resharding request can begenerated/provided (e.g., to system 120 and/or producer 410A),requesting that the referenced shard(s) (which may contain four or moremessages) be resharded (e.g., to include two messages, as shown).

At operation 562, a second shard is received (e.g., in response to therequest at operation 560). In certain implementations, such a shard caninclude a second shard version attribute. Additionally, in certainimplementations the referenced shard can be received/originate fromsystem 120 and/or producer 410A (as shown in FIG. 4B). In certainimplementations, such a second shard (e.g., shard 430E) can include asecond shard version attribute (e.g., attribute 458E, as shown in FIG.4B and described herein).

At operation 564, an operation is performed with respect to the secondshard (e.g., shard 430E as received at operation 562) in lieu ofperforming an operation with respect to the first shard (e.g., shard430D as received at operation 552). In certain implementations, such anoperation (e.g., with respect to shard 430E) can be performed based on adetermination that the current shard version (e.g., of the producer410A) is consistent with the second shard version attribute (e.g.,‘VERSION: 2’).

While many of the examples described herein are illustrated with respectto single server and/or individual devices, this is simply for the sakeof clarity and brevity. However, it should be understood that thedescribed technologies can also be implemented (in any number ofconfigurations) across multiple devices and/or other machines/services.

It should also be noted that while the technologies described herein areillustrated primarily with respect to stateless stream handling andresharding, the described technologies can also be implemented in anynumber of additional or alternative settings or contexts and towards anynumber of additional objectives. It should be understood that furthertechnical advantages, solutions, and/or improvements (beyond thosedescribed and/or referenced herein) can be enabled as a result of suchimplementations.

Certain implementations are described herein as including logic or anumber of components, modules, or mechanisms. Modules can constituteeither software modules (e.g., code embodied on a machine-readablemedium) or hardware modules. A “hardware module” is a tangible unitcapable of performing certain operations and can be configured orarranged in a certain physical manner. In various exampleimplementations, one or more computer systems (e.g., a standalonecomputer system, a client computer system, or a server computer system)or one or more hardware modules of a computer system (e.g., a processoror a group of processors) can be configured by software (e.g., anapplication or application portion) as a hardware module that operatesto perform certain operations as described herein.

In some implementations, a hardware module can be implementedmechanically, electronically, or any suitable combination thereof. Forexample, a hardware module can include dedicated circuitry or logic thatis permanently configured to perform certain operations. For example, ahardware module can be a special-purpose processor, such as aField-Programmable Gate Array (FPGA) or an Application SpecificIntegrated Circuit (ASIC). A hardware module can also includeprogrammable logic or circuitry that is temporarily configured bysoftware to perform certain operations. For example, a hardware modulecan include software executed by a general-purpose processor or otherprogrammable processor. Once configured by such software, hardwaremodules become specific machines (or specific components of a machine)uniquely tailored to perform the configured functions and are no longergeneral-purpose processors. It will be appreciated that the decision toimplement a hardware module mechanically, in dedicated and permanentlyconfigured circuitry, or in temporarily configured circuitry (e.g.,configured by software) can be driven by cost and time considerations.

Accordingly, the phrase “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. As used herein,“hardware-implemented module” refers to a hardware module. Consideringimplementations in which hardware modules are temporarily configured(e.g., programmed), each of the hardware modules need not be configuredor instantiated at any one instance in time. For example, where ahardware module comprises a general-purpose processor configured bysoftware to become a special-purpose processor, the general-purposeprocessor can be configured as respectively different special-purposeprocessors (e.g., comprising different hardware modules) at differenttimes. Software accordingly configures a particular processor orprocessors, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules can be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications can be achievedthrough signal transmission (e.g., over appropriate circuits and buses)between or among two or more of the hardware modules. In implementationsin which multiple hardware modules are configured or instantiated atdifferent times, communications between such hardware modules can beachieved, for example, through the storage and retrieval of informationin memory structures to which the multiple hardware modules have access.For example, one hardware module can perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module can then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules can also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein can beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors can constitute processor-implemented modulesthat operate to perform one or more operations or functions describedherein. As used herein, “processor-implemented module” refers to ahardware module implemented using one or more processors.

Similarly, the methods described herein can be at least partiallyprocessor-implemented, with a particular processor or processors beingan example of hardware. For example, at least some of the operations ofa method can be performed by one or more processors orprocessor-implemented modules. Moreover, the one or more processors canalso operate to support performance of the relevant operations in a“cloud computing” environment or as a “software as a service” (SaaS).For example, at least some of the operations can be performed by a groupof computers (as examples of machines including processors), with theseoperations being accessible via a network (e.g., the Internet) and viaone or more appropriate interfaces (e.g., an API).

The performance of certain of the operations can be distributed amongthe processors, not only residing within a single machine, but deployedacross a number of machines. In some example implementations, theprocessors or processor-implemented modules can be located in a singlegeographic location (e.g., within a home environment, an officeenvironment, or a server farm). In other example implementations, theprocessors or processor-implemented modules can be distributed across anumber of geographic locations.

The modules, methods, applications, and so forth described inconjunction with FIGS. 1A-5C are implemented in some implementations inthe context of a machine and an associated software architecture. Thesections below describe representative software architecture(s) andmachine (e.g., hardware) architecture(s) that are suitable for use withthe disclosed implementations.

Software architectures are used in conjunction with hardwarearchitectures to create devices and machines tailored to particularpurposes. For example, a particular hardware architecture coupled with aparticular software architecture will create a mobile device, such as amobile phone, tablet device, or so forth. A slightly different hardwareand software architecture can yield a smart device for use in the“internet of things,” while yet another combination produces a servercomputer for use within a cloud computing architecture. Not allcombinations of such software and hardware architectures are presentedhere, as those of skill in the art can readily understand how toimplement the inventive subject matter in different contexts from thedisclosure contained herein.

FIG. 6 is a block diagram illustrating components of a machine 600,according to some example implementations, able to read instructionsfrom a machine-readable medium (e.g., a machine-readable storage medium)and perform any one or more of the methodologies discussed herein.Specifically, FIG. 6 shows a diagrammatic representation of the machine600 in the example form of a computer system, within which instructions616 (e.g., software, a program, an application, an applet, an app, orother executable code) for causing the machine 600 to perform any one ormore of the methodologies discussed herein can be executed. Theinstructions 616 transform the general, non-programmed machine into aparticular machine programmed to carry out the described and illustratedfunctions in the manner described. In alternative implementations, themachine 600 operates as a standalone device or can be coupled (e.g.,networked) to other machines. In a networked deployment, the machine 600can operate in the capacity of a server machine or a client machine in aserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine 600 cancomprise, but not be limited to, a server computer, a client computer,PC, a tablet computer, a laptop computer, a netbook, a set-top box(STB), a personal digital assistant (PDA), an entertainment mediasystem, a cellular telephone, a smart phone, a mobile device, a wearabledevice (e.g., a smart watch), a smart home device (e.g., a smartappliance), other smart devices, a web appliance, a network router, anetwork switch, a network bridge, or any machine capable of executingthe instructions 616, sequentially or otherwise, that specify actions tobe taken by the machine 600. Further, while only a single machine 600 isillustrated, the term “machine” shall also be taken to include acollection of machines 600 that individually or jointly execute theinstructions 616 to perform any one or more of the methodologiesdiscussed herein.

The machine 600 can include processors 610, memory/storage 630, and I/Ocomponents 650, which can be configured to communicate with each othersuch as via a bus 602. In an example implementation, the processors 610(e.g., a Central Processing Unit (CPU), a Reduced Instruction SetComputing (RISC) processor, a Complex Instruction Set Computing (CISC)processor, a Graphics Processing Unit (GPU), a Digital Signal Processor(DSP), an ASIC, a Radio-Frequency Integrated Circuit (RFIC), anotherprocessor, or any suitable combination thereof) can include, forexample, a processor 612 and a processor 614 that can execute theinstructions 616. The term “processor” is intended to include multi-coreprocessors that can comprise two or more independent processors(sometimes referred to as “cores”) that can execute instructionscontemporaneously. Although FIG. 6 shows multiple processors 610, themachine 600 can include a single processor with a single core, a singleprocessor with multiple cores (e.g., a multi-core processor), multipleprocessors with a single core, multiple processors with multiples cores,or any combination thereof.

The memory/storage 630 can include a memory 632, such as a main memory,or other memory storage, and a storage unit 636, both accessible to theprocessors 610 such as via the bus 602. The storage unit 636 and memory632 store the instructions 616 embodying any one or more of themethodologies or functions described herein. The instructions 616 canalso reside, completely or partially, within the memory 632, within thestorage unit 636, within at least one of the processors 610 (e.g.,within the processor's cache memory), or any suitable combinationthereof, during execution thereof by the machine 600. Accordingly, thememory 632, the storage unit 636, and the memory of the processors 610are examples of machine-readable media.

As used herein, “machine-readable medium” means a device able to storeinstructions (e.g., instructions 616) and data temporarily orpermanently and can include, but is not limited to, random-access memory(RAM), read-only memory (ROM), buffer memory, flash memory, opticalmedia, magnetic media, cache memory, other types of storage (e.g.,Erasable Programmable Read-Only Memory (EEPROM)), and/or any suitablecombination thereof. The term “machine-readable medium” should be takento include a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storethe instructions 616. The term “machine-readable medium” shall also betaken to include any medium, or combination of multiple media, that iscapable of storing instructions (e.g., instructions 616) for executionby a machine (e.g., machine 600), such that the instructions, whenexecuted by one or more processors of the machine (e.g., processors610), cause the machine to perform any one or more of the methodologiesdescribed herein. Accordingly, a “machine-readable medium” refers to asingle storage apparatus or device, as well as “cloud-based” storagesystems or storage networks that include multiple storage apparatus ordevices. The term “machine-readable medium” excludes signals per se.

The I/O components 650 can include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/Ocomponents 650 that are included in a particular machine will depend onthe type of machine. For example, portable machines such as mobilephones will likely include a touch input device or other such inputmechanisms, while a headless server machine will likely not include sucha touch input device. It will be appreciated that the I/O components 650can include many other components that are not shown in FIG. 6 . The I/Ocomponents 650 are grouped according to functionality merely forsimplifying the following discussion and the grouping is in no waylimiting. In various example implementations, the I/O components 650 caninclude output components 652 and input components 654. The outputcomponents 652 can include visual components (e.g., a display such as aplasma display panel (PDP), a light emitting diode (LED) display, aliquid crystal display (LCD), a projector, or a cathode ray tube (CRT)),acoustic components (e.g., speakers), haptic components (e.g., avibratory motor, resistance mechanisms), other signal generators, and soforth. The input components 654 can include alphanumeric inputcomponents (e.g., a keyboard, a touch screen configured to receivealphanumeric input, a photo-optical keyboard, or other alphanumericinput components), point based input components (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, or another pointinginstrument), tactile input components (e.g., a physical button, a touchscreen that provides location and/or force of touches or touch gestures,or other tactile input components), audio input components (e.g., amicrophone), and the like.

In further example implementations, the I/O components 650 can includebiometric components 656, motion components 658, environmentalcomponents 660, or position components 662, among a wide array of othercomponents. For example, the biometric components 656 can includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram basedidentification), and the like. The motion components 658 can includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The environmental components 660 can include, for example, illuminationsensor components (e.g., photometer), temperature sensor components(e.g., one or more thermometers that detect ambient temperature),humidity sensor components, pressure sensor components (e.g.,barometer), acoustic sensor components (e.g., one or more microphonesthat detect background noise), proximity sensor components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gasdetection sensors to detect concentrations of hazardous gases for safetyor to measure pollutants in the atmosphere), or other components thatcan provide indications, measurements, or signals corresponding to asurrounding physical environment. The position components 662 caninclude location sensor components (e.g., a Global Position System (GPS)receiver component), altitude sensor components (e.g., altimeters orbarometers that detect air pressure from which altitude can be derived),orientation sensor components (e.g., magnetometers), and the like.

Communication can be implemented using a wide variety of technologies.The I/O components 650 can include communication components 664 operableto couple the machine 600 to a network 680 or devices 670 via a coupling682 and a coupling 672, respectively. For example, the communicationcomponents 664 can include a network interface component or othersuitable device to interface with the network 680. In further examples,the communication components 664 can include wired communicationcomponents, wireless communication components, cellular communicationcomponents, Near Field Communication (NFC) components, Bluetooth®components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and othercommunication components to provide communication via other modalities.The devices 670 can be another machine or any of a wide variety ofperipheral devices (e.g., a peripheral device coupled via a USB).

Moreover, the communication components 664 can detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 664 can include Radio Frequency Identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information can be derived via the communication components664, such as location via Internet Protocol (IP) geolocation, locationvia Wi-Fi® signal triangulation, location via detecting an NFC beaconsignal that can indicate a particular location, and so forth.

In various example implementations, one or more portions of the network680 can be an ad hoc network, an intranet, an extranet, a virtualprivate network (VPN), a local area network (LAN), a wireless LAN(WLAN), a WAN, a wireless WAN (WWAN), a metropolitan area network (MAN),the Internet, a portion of the Internet, a portion of the PublicSwitched Telephone Network (PSTN), a plain old telephone service (POTS)network, a cellular telephone network, a wireless network, a Wi-Fi®network, another type of network, or a combination of two or more suchnetworks. For example, the network 680 or a portion of the network 680can include a wireless or cellular network and the coupling 682 can be aCode Division Multiple Access (CDMA) connection, a Global System forMobile communications (GSM) connection, or another type of cellular orwireless coupling. In this example, the coupling 682 can implement anyof a variety of types of data transfer technology, such as SingleCarrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized(EVDO) technology, General Packet Radio Service (GPRS) technology,Enhanced Data rates for GSM Evolution (EDGE) technology, thirdGeneration Partnership Project (3GPP) including 3G, fourth generationwireless (4G) networks, Universal Mobile Telecommunications System(UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability forMicrowave Access (WiMAX), Long Term Evolution (LTE) standard, othersdefined by various standard-setting organizations, other long rangeprotocols, or other data transfer technology.

The instructions 616 can be transmitted or received over the network 680using a transmission medium via a network interface device (e.g., anetwork interface component included in the communication components664) and utilizing any one of a number of well-known transfer protocols(e.g., HTTP). Similarly, the instructions 616 can be transmitted orreceived using a transmission medium via the coupling 672 (e.g., apeer-to-peer coupling) to the devices 670. The term “transmissionmedium” shall be taken to include any intangible medium that is capableof storing, encoding, or carrying the instructions 616 for execution bythe machine 600, and includes digital or analog communications signalsor other intangible media to facilitate communication of such software.

Throughout this specification, plural instances can implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations can be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationscan be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component can beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Although an overview of the inventive subject matter has been describedwith reference to specific example implementations, variousmodifications and changes can be made to these implementations withoutdeparting from the broader scope of implementations of the presentdisclosure. Such implementations of the inventive subject matter can bereferred to herein, individually or collectively, by the term“invention” merely for convenience and without intending to voluntarilylimit the scope of this application to any single disclosure orinventive concept if more than one is, in fact, disclosed.

The implementations illustrated herein are described in sufficientdetail to enable those skilled in the art to practice the teachingsdisclosed. Other implementations can be used and derived therefrom, suchthat structural and logical substitutions and changes can be madewithout departing from the scope of this disclosure. The DetailedDescription, therefore, is not to be taken in a limiting sense, and thescope of various implementations is defined only by the appended claims,along with the full range of equivalents to which such claims areentitled.

As used herein, the term “or” can be construed in either an inclusive orexclusive sense. Moreover, plural instances can be provided forresources, operations, or structures described herein as a singleinstance. Additionally, boundaries between various resources,operations, modules, engines, and data stores are somewhat arbitrary,and particular operations are illustrated in a context of specificillustrative configurations. Other allocations of functionality areenvisioned and can fall within a scope of various implementations of thepresent disclosure. In general, structures and functionality presentedas separate resources in the example configurations can be implementedas a combined structure or resource. Similarly, structures andfunctionality presented as a single resource can be implemented asseparate resources. These and other variations, modifications,additions, and improvements fall within a scope of implementations ofthe present disclosure as represented by the appended claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

1.-20. (canceled)
 21. A system comprising: a processing device; and amemory coupled to the processing device and storing instructions that,when executed by the processing device, cause the system to performoperations comprising: receiving a first shard comprising one or moremessages; receiving the first state attribute, wherein the first stateattribute comprises a token assigned based on a processing capacity of astreaming system associating the first shard with a first stateattribute; providing the first shard and the first state attribute as anatomic update within a data stream; and initiating an adjustment of anoperation of a message production source based on the received firststate attribute assigned based on a processing capacity of a streamingsystem.
 22. The system of claim 21, wherein the memory further storesinstructions to cause the system to perform operations comprisingrequesting the first state attribute from the first shard.
 23. Thesystem of claim 22, wherein the memory further stores instructions tocause the system to perform operations comprising: receiving the firststate attribute; and providing a second shard within the data streambased on the received first state attribute.
 24. The system of claim 21,wherein the first state attribute reflects an importance of one or moreof the messages.
 25. The system of claim 21, wherein the first stateattribute reflects a location of one or more of the messages.
 26. Thesystem of claim 21, wherein an operation associated with the first shardis performed based on the first state attribute.
 27. The system of claim21, wherein the atomic update comprises a plurality of updates that arecollectively performed or rejected.
 28. The system of claim 21, whereinproviding the first shard and the first state attribute comprisesproviding the first shard and the first state attribute as a conditionalupdate within the data stream.
 29. The system of claim 21, wherein thefirst state attribute comprises a first sequence identifier.
 30. Amethod comprising: receiving a first shard comprising one or moremessages; associating the first shard with a first state attribute; andproviding the first shard and the first state attribute as an updatewithin a data stream.
 31. The method of claim 30, further comprisingrequesting the first state attribute from the first shard.
 32. Themethod of claim 30, further comprising: receiving the first stateattribute; and providing a second shard within the data stream based onthe received first state attribute.
 33. The method of claim 30, whereinthe first state attribute reflects an importance of one or more of themessages.
 34. The method of claim 30, wherein the first state attributereflects a location of one or more of the messages.
 35. The method ofclaim 30, wherein providing the first shard and the first stateattribute comprises providing the first shard and the first stateattribute as an atomic update within the data stream.
 36. The method ofclaim 30, wherein providing the first shard and the first stateattribute comprises providing the first shard and the first stateattribute as a conditional update within the data stream.
 37. The methodof claim 30, wherein the first state attribute comprises a firstsequence identifier.
 38. A non-transitory computer readable mediumhaving instructions stored thereon that, when executed by a processingdevice, cause the processing device to perform operations comprising:receiving a first shard comprising one or more messages; associating thefirst shard with a first sequence identifier; associating the firstshard with a first state attribute that reflects an importance of one ormore of the messages; and providing the first shard and the first stateattribute as an update within a data stream; requesting the first stateattribute from the first shard; receiving the first state attribute; andproviding a second shard within the data stream based on the receivedfirst state attribute.